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Abstract 



The Service Level Agreement (SLA) based grid su- 
perscheduling approach promotes coordinated resource 
sharing. Superscheduling is facilitated between admin- 
istratively and topologically distributed grid sites by grid 
schedulers such as Resource brokers. In this work, we 
present a market-based SLA coordination mechanism. 
We based our SLA model on a well known contract net 
protocol. 

The key advantages of our approach are that it al- 
lows: (i) resource owners to have finer degree of con- 
trol over the resource allocation that was previously not 
possible through traditional mechanism; and (ii) super- 
schedulers to bid for SLA contracts in the contract net 
with focus on completing the job within the user speci- 
fied deadline. In this work, we use simulation to show 
the effectiveness of our proposed approach. 

1 Introduction 

A grid 1 23 1 enables aggregation of topologically dis- 
tributed scientific instruments, storage structure, data, 
applications and computational resources. The notion 
of Grid computing is well beyond the traditional Par- 
allel and Distributed Computing Systems (PDCS) as it 
involves various resources that belong to different ad- 
ministrative domains and are controlled by domain spe- 
cific resource management policies. Furthermore, the 
grids have evolved around complex business and service 
models where various small sites (resource owners) col- 
laborate for computational and economic benefits. The 
task of resource management and application schedul- 
ing over a grid is a complex undertaking due to resource 
heterogeneity, domain specific policy, dynamic environ- 
ment, and various socio-economic and political factors. 



The Grid superscheduling |37| problem is defined 
as: "scheduling jobs across the grid resources such as 
computational clusters, parallel supercomputers, desk- 
top machines that belong to different administrative do- 
mains". Superscheduling in computational grids is fa- 
cilitated by specialized superschedulers such as Grid 
Federation Agent (33), NASA-Superscheduler l38l . 
Nimrod-G |6|, and Condor-G |24|. Superscheduling 
activity involves (i) querying grid resource information 
services (GRIS) EI]|27][T9]|3i>||8] for locating resources 
that match the job requirements; (ii) coordinating and 
negotiating SLAs; and (iii) job scheduling. The grid 
resources are managed by their local resource manage- 
ment systems (LRMSes) such as Condor (29), PBS GDI, 
SGE g6|, Legion EJ, Alchemi |30) and LSF E). The 
LRMSes manage job queues, initiate and monitor their 
execution. 

In this work, we propose a SLA 1321 1201 [18 1 based 
coordinated superscheduling scheme for federated grid 
systems. An SLA is the agreement negotiated between 
a superscheduler (resource consumer) and LRMSes (re- 
source provider) about acceptable job QoS constraints. 
These QoS constraints may include the job response 
time and budget spent. Inherently, a SLA is the guar- 
antee given by a resource provider to a remote site job 
superscheduler for completing the job within the spec- 
ified deadline or within the agreed budget or satisfy- 
ing both at the same time. A SLA-based coordinated 
job superscheduling approach has many advantages: (i) 
It inhibits superschedulers from submitting unbounded 
amount of work to LRMSes; (ii) once a SLA is reached, 
users' are certain that agreed QoS shall be delivered by 
the system; (iii) job queuing or processing delay is sig- 
nificantly reduced thus leading to enhanced QoS, else a 
penalty model [48 1 is applied to compensate them ; and 
(iv) gives LRMSes more autonomy and better control 
over resource allocation decisions. 

Our SLA model incorporates an economic mecha- 
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nism 1221 1121 |5J |5] E) for job superscheduling and re- 
source allocation. The economic mechanism enables 
the regulation of supply and demand of resources, offers 
incentive to the resource owners for leasing, and pro- 
motes QoS based resource allocation. We mainly focus 
on the decentralized commodity market model 1 42) . In 
this model every resource has a price, which is based on 
the demand, supply and value. An economy driven re- 
source allocation methodology focuses on: (i) optimiz- 
ing resource provider's payoff function; and (ii) increas- 
ing end-user's perceived QoS value. Note that our pro- 
posed superscheduling approach is studied as part of a 
new and emerging grid system which we call as Grid- 
Federation 1 33 1 . General details about this system can 
be found in section[2] 

Our SLA model considers a collection of compu- 
tational cluster resources as a contract net |41|. As 
job arrives, the grid superschedulers undertake one-to- 
one contract negotiation with the LRMSes managing the 
concerned resource. The SLA contract negotiation mes- 
sage includes: (i) whether a job can be completed within 
the specified deadline; and (ii) SLA bid expiration time 
(maximum amount of time a superscheduler is willing 
to wait before finalizing SLA). The SLA bid expiration 
time methodology we apply here is different from that 
adopted in the Tycoon system |28 1. In Tycoon, the SLA 
bid expiration time at a resource is the same for all the 
jobs irrespective of their size or deadline. In this case, 
the total bid-processing delay is directly controlled by 
the local resource auctioneer. In our model, the super- 
scheduler bids with a SLA bid expiration time propor- 
tional to the job's deadline. The focus is on meeting the 
job's SLA requirements, in particular the job deadline. 
The SLA contract negotiation in NASA-Superscheduler 
and Condor-Flock P2P 1 1 1 1 is based on general broad- 
cast and limited broadcast communication mechanism 
respectively. Hence, these approaches have the follow- 
ing limitations: (i) high network overhead; and (ii) scal- 
ability problems. 

Our time constrained SLA bid-based contract ne- 
gotiation approach gives LRMSes finer control over 
the resource allocation decision as compared to 
traditional First-Come-First-Serve (FCFS approach). 
Existing superscheduling systems including NASA- 
Superscheduler, Condor-Flock P2P, Nimrod-G, Condor- 
G and Legion-Federation |46| assumes every LRMS al- 
locates the resources using FCFS scheduling scheme. 
In this work, we propose a Greedy backfilling LRMS 
scheduling that focus on maximizing resource owner's 
payoff function. In this case, a LRMS maintains a queue 
of SLA bid requests generated by various supersched- 
ulers in the system at a time t. Every SLA bid has an 
associated expiry time with it. If the concerned LRMS 
does not reply within that period, then the SLA request 



is considered as expired. Greedy backfilling is based 
on well known Greedy or Knapsack method I25lll7l l9l. 
The LRMSes periodically iterates through the local SLA 
bids and finalizes the contract with those that fit the re- 
source owner's payoff function. 

The main contribution of this work includes (ii) SLA 
bid based superscheduling approach; (ii) a Greedy back- 
filling cluster scheduling approach for LRMSes that fo- 
cus on maximizing the resource owners' payoff func- 
tion; and (iii) allowing resource owners to have finer de- 
gree of control over resource allocation decisions. In 
this work, we use simulation to evaluate the feasibility 
of our proposed approach. 



Table 1 . Notations 



Symbol 


Meaning 




number ofGFAs. 




resource access cost al GFA i. 


Pi 


number oi ro'tv w;-, ,n GFA i. 




z-lh job from the j-lh user of fc-lh GFA. 


Pi.i.k 


number of processor required by ■ ■ ^ . 


b i,i,k 


assigned budget to J% j ^ . 


*i.i.H 


assigned deadline Lo .7^ j ^ . 




effective deadline for .7^ j ^ . 




lime function ('expected response lime for ,/ ; - j ^ al resource k). 




cost funelion (expected budge! spenl Tor J .- j f, at resource k). 


H 


incenlive earned by resource uv. net /■■ u\vr simulation period. 




returns next SLA bid inlerval Af,,, P n . . , for J ■ ■ ... 


tne H,.j,k 


total SLA bid interval/delay for J ^ j ^. 


Q m ,t 


sel of jobs lhat have been assigned bin mu accepted al GFA m at lime 
t. 




set of jobs thai have been accepted al GFA m al lime t. 




sel of jobs sorted in decreasing order of incentive it provides lo the 
resource owner al GFA m al lime t. 




number ol u>ers over all clusters (J^ ^ n^.n^ number of users 
at GFA k). 


n 3 


total jobs in the federation u ) — i n h' 


ts i i h 


job submission delay ('user lo GFA). 




finished job return delay ( GFA to user) . 




total delay for p-lh SLA bid for J ; j fc . 


X SLAi 


SLA arrival rale al GFA i. 




SLA satisfaction rale al GFA i. 




job length for .■ ^ (in terms of million instructions) 




communication overhead for .7.^ j ^ 



1.1 Organization of this paper 



The paper is organized as follows. In section |2] 
we present a brief overview of Grid-Federation super- 
scheduling framework. Section 13.11 presents details 
about our proposed bid-based SLA contract negotiation 
model. In section l3~2l we give details about our pro- 
posed Greedy backfilling LRMS scheduling approach. 
In section|4] we present various experiments and discuss 
our results. Section[5]presents some of the related work 
in superscheduling. We end this paper with concluding 
remarks and our future vision in Section[6] Note that, in 
Table. [Qwe define various notations that we use in this 
paper. 
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2 Brief overview of Grid- Federation 

The Grid-Federation 1 33 1 system is defined as a large 
scale resource sharing system that consists of a cooper- 
ative federation of distributed clusters based on policies 
defined by their owners (shown in Fig.l). Fig.l shows 
an abstract model of our Grid-Federation over a shared 
federation directory. To enable policy based transparent 
resource sharing between these clusters, we define and 
model a new RMS system, which we call Grid Feder- 
ation Agent (GFA). Currently, we assume that the di- 
rectory information is shared using some efficient pro- 
tocol (e.g. a peer-to-peer protocol (2] QH |SJ fT4l l. 
In this case, the P2P system provides a decentralized 
database with efficient updates and range query capa- 
bilities. Individual GFAs access the directory informa- 
tion using the interface shown in Fig.l, i.e. subscribe, 
quote, unsubscribe, and query. The specifics of the in- 
terface can be found in 1 34 1 . Our approach considers the 
emerging computational economy metaphor |6| 1451 
for the Grid-Federation. Some of the commonly used 
economic models 1121 in resource allocation includes 
the commodity market model, the posted price model, 
the bargaining model, the tendering/contract-net model, 
the auction model, the bid-based proportional resource 
sharing model, the community/coalition model and the 
monopoly model. Grid-Federation considers decen- 



tralized commodity market model for managing job 
scheduling and resource allocation. In this case, the re- 
source owners: (i) can clearly define what is shared in 
the Grid-Federation while maintaining a complete au- 
tonomy; (ii) can dictate who is given access; and (iii) 
get incentives for leasing their resources to federation 
users. 



In Fig.l a user who is local to GFA 3 is submitting 
a job. If the user's job QoS can't be satisfied locally 
then GFA 3 queries the federation directory to obtain the 
quote of the 1-st fastest (if the user is seeking optimize 
for time (OFT)) or 1-st cheapest cluster ( if the user is 
seeking optimize for cost (OFC)). In this case, the fed- 
eration directory returns the quote advertised by GFA 2. 
Following this, GFA 3 bids for SLA contract (enquiry 
about QoS guarantee in terms of response time) at GFA 
2. If GFA has too much load or the SLA bid does not fit 
the resource owner payoff function, the bid eventually 
timeouts. In this case, the SLA bid by GFA 2 timeouts. 
As next superscheduling iteration, GFA 3 queries the 
federation directory for the 2-nd cheapest/fastest GFA 
and so on. The process of SLA bids is repeated until 
GFA 3 finds a GFA that can schedule the job (i.e. accept 
the SLA bid) (in this example the job is finally scheduled 
on cluster 4). 
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3 Models 
3.1 SLA model 

The SLA model we consider is that of a set of dis- 
tributed cluster resources each offering a fixed amount 
of processing power. The resources form part of the 
federated grid environment and are shared among the 
end-users each having its own SLA parameters. SLAs 
are managed and coordinated through admission control 
mechanism enforced by GFA at each resource site. Each 
user in the federation has a job Jij,k- We write Jij,k 
to represent the z-th job from the j-th user of the fe-th 
resource. A job consists of the number of processors re- 
quired, Pij,k, the job length, li,j,k ( m terms of million 
instructions), the communication overhead, a.i t j,k and 
SLA parameters the budget, fo;.j,fc, the deadline or maxi- 
mum delay, d% j k- More details about the job model can 
be found in 1 33l . 

3.1.1 SLA bid with expiration time (based on con- 
tract net protocol 1 41 1) 

The collection of GFAs in the federation are referred to 
as a contract net and job-migration in the net is facil- 
itated through the SLA contracts. Each GFA can take 
on two roles either a manager or contractor. The GFA 
to which a user submits a job for processing is referred 
as the manager GFA. The manager GFA is responsible 
for superscheduling the job in the net. The GFA which 
accepts the job from the manager GFA and overlooks its 
execution is referred to as the contractor GFA. Individual 
GFAs are not assigned these roles in advance. The role 
may change dynamically over time as per the user's job 
requirement. Thus, the GFA alternates between these 
two roles or adheres to both over the course of super- 
scheduling. 

As jobs arrive to a GFA, the GFA adopts the role of a 
manager. Following this, the manager GFA queries the 
shared federation directory to obtain the quote for the 
contractor GFA that matches the user specified SLA pa- 
rameters. Note that, users can seek optimization for one 
of the SLA parameters i.e. either response time (OFT) 
or the budget spent (OFC). Once, the manager obtains 
the quote of the desired contractor, it undertakes one-to- 
one SLA contract negotiation with the contractor. The 
SLA contract negotiation message includes: (i) whether 
the job Jj,j,fc can be completed within the specified 
deadline; and (ii) SLA bid expiration time At neg ij^,i- 
The contractor GFA has to reply within the bid time 
At n egi.j,k,i, else the managerGFA undertakes SLA con- 
tract negotiation with the next available contractor in the 
net. Algorithm SLA bidding mechanism (refer to Algo- 



rithm 1 ) depicts various events and corresponding super- 
scheduling actions undertaken by a GFA. 

Our SLA contract model considers a part of the total 
job deadline as the SLA contract negotiation time (refer 
to Eq. [TJ. The manager GFA bids with different SLA 
expiration interval given by Eq.El InFig.EI we show the 
job superscheduling timeline. The timeline includes the 
job submission delay, t s . . k , total SLA contract negoti- 
ation delay, t negt . k , expected response time (computed 
using Eq.[0 and finished job return delay, t Ti k . The to- 
tal SLA contract bidding delay available to the manager 
GFA for superscheduling job J%,j,k is given by 

tne9i.j,k ~ di,j,k ~ ts i jtk ~ — ^r iJik (1) 

The total SLA contract bid negotiation delay 
t negi ■ k assumes a finite number of values At negi . kl , 
^*ne 9< , j , fe , 2 ,-,Atne 9< , 3 -, fe , n in superscheduling a job 
Ji j k (refer to Fig. [2). We define the value of At neg . k , 
by' 

^neg idtkJ — ^ , i > U 

(2) 

Note that, the value for At negi . k l can be given by 
other distributions Q such as uniform or random. We 
intend to analyze various distributions for SLA bid inter- 
val and study its effect on our proposed superscheduling 
approach in our future work. For simplicity, in this work 
we use the distribution given by Eq.|2] 

As the superscheduling iteration increases, the man- 
ager GFAs give less time to the contractor to decide on 
the SLA in order to meet the user's job deadline. This 
approach gives large number of scheduling iteration to 
the manager GFA. However, if the user's SLA parame- 
ters cannot be satisfied (after iterating up to the greatest 
r such that GFA could feasibly complete the job) then 
the job is dropped. To summarize, a SLA bid for job 
Jij.k includes 

• Z-th SLA bid expiry interval t neg . . k , (computed 
using Eq.|2j; 

• expected response time (dfj k ) (computed using 
Eq.Q). 

We consider the function 

t ■ Ji,j,k — > z+ (3) 

which returns the next allowed SLA bidding time in- 
terval At ne g i jtk p for a job using EqE] 
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effective deadline 



H,j,k 



Figure 2. Job superscheduling timeline 



Algorithm 1: SLA bidding mechanism 



0.1 PROCEDURE: SLA_BIDDING_MECHANISM 
0.2 begin 
03 begin 

0.4 SUB-PROCEDURE: 

EVENT _USER_JOB_SUBMIT (J 4j , fc ) 



0.5 
0.6 
0.7 
0.8 
0.9 



call SLA3ID ( J; k ). 



end 
begin 



SUB-PROCEDURE: SLA3ID (J ijifc ) 
Send SLA bid for job Ji.j.u to the next available 
contractor GFA (obtained by querying the shared 
federation directory). 



0.10 


end 


0.11 


begin 


0.12 




SUB-PROCEDURE: 

EVENT _SLA_B ID .REPLY (J itjik ) 


0.13 




if SLA Contract Accepted then 


0.14 




| Send the job J, j to accepting 


0.15 




end 


0.16 




else 


0.17 




| call SLA_BID_TIMEOUT (Jjj 


0.18 




end 


0.19 


end 


0.20 


begin 


0.21 




SUB-PROCEDURE: 
SLA_BID_TIMEOUT(J i d , k ) 


0.22 




if r (Jij t k) > then 


0.23 




| call'sLA_BID(J ijifc ). 


0.24 




end 


0.25 




else 


0.26 




| Drop the job Ji,j,fc. 


0.27 




end 


0.28 


end 


0.29 end 





3.2 Greedy backfilling: 
model) 



(LRMS scheduling 



Most of the existing LRMSes apply system-centric 
policies for allocating jobs to the resources. Some of the 
well known system-centric policies include: (i) FCFS; 
(ii) Conservative backfilling 1421 : and (ii) Easy backfill- 
ing |21|. Experiments |35| have shown that job back- 
filling approach offers significant improvement in per- 
formance over FCFS scheme. However, these system 
centric approaches allocate resource based on parame- 



ters that enhance system utilization or throughput. The 
LRMS either focuses on minimizing the response time 
(sum of queue time and actual execution time) or max- 
imizing overall resource utilization of the system and 
these are not specifically applied on a per-user basis 
(user oblivious). Further, the system centric LRMSes 
treat all resources with the same scale, thus neglecting 
the resource owner payoff function. In this case, the re- 
source owners do not have any control over resource al- 
location decisions. While in reality the resource owner 
would like to dictate how his resources are made avail- 
able to the outside world and apply a resource allocation 
policy that suits his payoff function. To summarize, the 
system-centric approaches do not provide mechanisms 
for resource owners to dictate resource: (i) sharing; (ii) 
access and ; (iii) allocation policies. 

To address this, we propose a Greedy method based 
resource allocation heuristic for LRMSes. Our proposed 
heuristic focuses on maximizing payoff function for the 
resource owners. The heuristic is based on the well 
known Greedy method |9 T7). The Greedy method for 
solving optimization problems considers greedily max- 
imizing or minimizing the short-term goals and hop- 
ing for the best without regard to the long-term effects. 
This method has been used to solve the knapsack prob- 
lem 1251 . Given a set S, of n items, with each item 
i, having a positive benefit hi, a positive weight Wi, 
knapsack capacity W and amount Xi, we consider for 
each item i. The Greedy heuristic focuses on max- 
imizing total benefit J2ies a ^i{ x i/ w i) w i tn constraint 
E ie s» %i < W, such that S a C S. 

Fig|3] shows the queue of SLA bids at each site in 
the federation. Every incoming SLA bid is added to the 
LRMS request queue, Q m ^ and a bid expiration timeout 
event is scheduled after time interval T(Jij^)- Every 
resource i has different SLA bid arrival rate, As and 
SLA bid satisfaction rate, [isLAi ■ The LRMS scheduler 
iterates through the SLA bid queue in case any of the 
following events occur: (i) new SLA bid arrives to the 
site; (ii) job completion; or (iii) SLA bid reaches its 
expiration time. Procedure Greedy backfilling (refer to 
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SLA bids (A SiAl ) SLA bids (A SLA2 ) SLA bids <.X S LA 3 ) SLA bids (X S LA„) 



Figure 3. SLA bid queues in Grid-Federation 



Algorithm 2) depicts various events and corresponding 
scheduling actions undertaken by the LRMS. 

Integer linear programming (ILP) formulation of 
scheduling heuristic 

Queue, Q m .t, maintains the the set of job SLA bids 
currently negotiated with the LRMS at GFA m by time 
t. We consider the SLA bid acceptance variable x^j^ 

Definition of variable 

%i,j,k — 1 if the SLA request for job Ji.j : k is ac- 
cepted; 

Xi j,fc = otherwise. 

The Greedy-Backfilling heuristic accepts SLA 
requests constrained to the availability of number 
of processors requested for job Ji,j,k and expected 
response time d| • k . 

Definition of the constraints 

l<i<rij 
1< ;<;/„ 
l<k<n 

p m total number of processors available at a 
LRMS (GFA) m. Pi,j,k denotes number of processor 
requested by the SLA bid for job Jij.k- All the accepted 
SLA bids for jobs are maintained in the queue Qf n t . 



Payoff or Objective function 



The LRMS scheduler accepts SLA bids for the 
jobs such that it maximizes the resource owners' payoff 
function by applying Greedy backfilling heuristic 

I m = max( ^ B {Ji^,k,Rm)) (5) 

l<i<nj 
l</<»„ 

l<fc<7! 

1< m<n 

3.3 Economic parameters 
3.3.1 Setting price (a) 

The resource owners configure the resource access cost 
Cj to reflect its demand in the federation. A resource 
owner can vary Cj depending on resource demand As LAi 
and resource supply fig LAi pattern. In case, As LAi > 
[is LAi, men me resource owner can increase the Cj. 
However, Asla ; depends on the user population pro- 
file. If the majority of users are seeking optimization 
for response time then time-efficient resources may in- 
crease Cj until AslA; = [i-SLAi - Furthermore, to find the 
bounds of Cj it is mandatory to consider the amount of 
budget available to the users. 

For simplicity, in this work we assume that Cj re- 
mains static throughout the simulations. We intend to 
analyze different pricing algorithm 111 61 l39l 1471 based 
on supply and demand function as a future work. Us- 
ing the static price Cj, we quantify how varying the SLA 
bid time affects the federated superscheduling systems' 
performance. In simulations, we configure Cj using the 
function: 

Ci = /(Mi) (6) 

where, 

fQj-i) = ~ Mi ( 7 ) 
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Algorithm 2'. Greedy-Backfilling 



PROCEDURE: GREEDY BACKFILLING 
begin 

r — pm 
c = 
Qm,t <- <t> 

Q a m ,t <- 4> 
Q s m ,t *~ <P 

begin 

SUB-PROCEDURE:Event_SLA_Bid_ARRIVAL(J i j |fc ) 

A SLA request message for the job Ji 7 j r k mat arrives at a GFA Q mi t < — Qm,t U {Jij^} 
Schedule the SLA bid timeout event after r ( J i t j r k ) time units 
call STRICT .GREEDY () 

end 
begin 

SUB-PROCEDURE:Event_SLA_Bid_Timeout(J ij . jifc ) 
A SLA bid for job Jij^k that reaches timeout period 
if (r > Pi, jik andd° j k > D(J iijik , R m ))then 

| CallRESERVE(J,' j:) , fc ) 
end 
else 

| Reject the SLA bid for job Jij,k 
I Reset Q m>t <- Q m-t - {J;,^} 
end 

end 
begin 

SUB-PROCEDURE:EventJob.Finish(J ij:jifc ) 
Ajob Jij,k that finishes at a GFA Reset r = r + Pi,j,k 
call STRICT.GREEDYO 

end 
begin 

SUB-PROCEDURE: RESERVE( Ji t j, k ) 
Reserve Pij y k processors for the job Jij y k 

Reset r = r - Pi,j,k, Qm,t <— Qm,t - {Ji,j,k}, Qm,t <~ Qrn.t U {Ji.j.k} 

end 
begin 

SUB-PROCEDURE: STRICT.GREEDYO 
Reset c = 

Sort SLA bids in Q m; t in decreasing order of incentives and store in t 



Get next SLA bid for job Jtj,k from the list i 
if (r > Pi,j,k anddi j k > D(.J iJ: k,R^ 

| CallRESERVE(J, j:jifc ) 
end 
else 

if e < sizeof(Q B m t ) then 

j Iterate through step 1.39 
end 
end 



=c+l 



)J then 



end 



end 
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c is the access price and fi is the speed of the fastest 
resource in the Grid-Federation. Details about how users 
are charged on per job basis can be found in 1 33 1. 

3.3.2 User budget and deadline 

While our simulations in the next section use trace data 
for job characteristics, the trace data does not include 
user specified budgets and deadlines on a per job ba- 
sis. In order to study our proposed SLA model and su- 
perscheduling approach, we are forced to fabricate these 
quantities and we include the models here. 

For a user, j, we allow each job from that user to be 
given a budget, 

h,j,k = 2 B(Jij y k, Rk)- (8) 

In other words, the total budget of a user over simu- 
lation is unbounded and we are interested in computing 
the budget that is required to schedule all of the jobs. 

Also, we let the deadline for job i be 

d itjik = 3D(J ij!k ,R k ). (9) 

We assign three times the expected response time for 
the given job, as compared to expected response time 
on the originating resource. We use the multiplying 
constant as 3, for allowing the superschedulers ample 
time during SLA bidding. However, as a future work 
we intend to analyze how does the system performance 
changes when multiplying constant approaches 1 and in- 
finity. Details about the budget and time function can be 
found in ll33l . 

4 Experiments and observations 

4.1 Workload and resource methodology 

We performed trace based simulation to evaluate the 
effectiveness of our SLA based superscheduling ap- 
proach. The workload trace data was obtained from Q] . 
The trace contains real time workload of various re- 
sources/supercomputers that are deployed at the Cor- 
nell Theory Center (CTC SP2), Swedish Royal Insti- 
tute of Technology (KTH SP2), Los Alamos National 
Lab (LANL CM5), LANL Origin 2000 Cluster (Nir- 
vana) (LANL Origin), NASA Ames (NASA iPSC) and 
San-Diego Supercomputer Center (SDSC Par96, SDSC 
Blue, SDSC SP2) (See Table 0. The workload trace is 
a record of usage data for parallel jobs that were sub- 
mitted to various resource facilities. Every job arrives, 
is allocated one or more processors for a period of time, 
and then leaves the system. Furthermore, every job in 
the workload has an associated arrival time, indicating 
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Table 2. Workload and Resource Configuration 



Index 


Resource / 
Cluster Name 


Trace Date 


Processors 


MIPS 
(rat- 
ing) 


Jobs 


Quote(Price) 


NIC to Network 
Bandwidth (Gb/Sec) 


1 


CTC SP2 


June96-May97 


512 


850 


79,302 


4.84 


2 


2 


KTH SP2 


Sep96-Aug97 


100 


900 


28,490 


5.12 


1.6 


3 


LANL CM5 


Oct94-Sep96 


1024 


700 


201,387 


3.98 


1 


4 


LANL Origin 


Nov99-Apr2000 


2048 


630 


121,989 


3.59 


1.6 


5 


NASA iPSC 


Oct93-Dec93 


128 


930 


42,264 


5.3 


4 


6 


SDSC Par96 


Dec95-Dec96 


416 


710 


38,719 


4.04 


1 


7 


SDSC Blue 


Apr2000-Jan2003 


1152 


730 


250,440 


4.16 


2 


8 


SDSC SP2 


Apr98-Apr2000 


128 


920 


73,496 


5.24 


4 



when it was submitted to the scheduler for considera- 
tion. As the experimental trace data does not include 
details about the network communication overhead in- 
volved for different jobs, we artificially introduced the 
communication overhead element as 10% of the total 
parallel job execution time. More details about the job 
execution time modeling can be found in 1 33 1. The sim- 
ulator was implemented using GridSim [ 13 1 toolkit that 
allows modeling and simulation of distributed system 
entities for evaluation of scheduling algorithms. To en- 
able the parallel workload simulation with GridSim, we 
extended the existing GridSim's Alloc Policy and Space 
Shared entities. 

Our simulation environment models the following 
basic entities in addition to existing entities in GridSim: 

• local user population - models the workload ob- 
tained from trace data; 

• GFA - generalized RMS system; 

• GFA queue - placeholder for incoming jobs from 
local user population and the federation; 

• GFA shared federation directory - simulates an ef- 
ficient distributed query process such as peer-to- 
peer. 

For evaluating the SLA based superscheduling, we 
assigned a synthetic QoS specification to each resource 
including the Quote value (price that a cluster owner 
charges for service), with varying MIPS rating and un- 
derlying network communication bandwidth. The sim- 
ulation experiments were conducted by utilizing work- 
load trace data over the total period of four days (in 
simulation units) at all the resources. We consider the 
following resource sharing environment for our experi- 
ments: 

• federation with economy - Experiments 1 and 2. 



4.2 Experiment 1 - Quantifying scheduling pa- 
rameters related to resource owners and 
end-users with varying total SLA bid time 

We quantify the following scheduling parameters re- 
lated to resource owners and end-users 

• resource owner: payoff function (total earnings, 
earnings per processor), resource utilization (in 
terms of total MI executed); 

• end-users: QoS satisfaction (average response 
time, average budget spent), number of jobs ac- 
cepted. 

We performed the simulations which comprised of 
end-users seeking OFT for their jobs (i.e. 100% users 
seek OFT). We vary the total SLA bid from 0% to 50% 
of total allowed job deadline. In case, no SLA bid de- 
lay is allowed (i.e. 0% of total allowed deadline) then 
the contacted GFA has to instantly make the admission 
control decision. In this case, we simulate FCFS based 
strategy for finalizing SLA. However, in other cases we 
consider Greedy backfilling SLA approach. 

4.3 Experiment 2 - Quantifying message com- 
plexity involved with varying total SLA bid 
time 

In this experiment we consider the message complex- 
ity involved in our proposed superscheduling approach. 
In this work, we consider the following superscheduling 
parameters related to overall system message complex- 
ity: 

• average message per job at a resource i: the number 
of SLA bid requests undertaken at a resource on the 
average before the job was actually scheduled ; 

• local message count: number of SLA bid schedul- 
ing messages undertaken for local jobs at a resource 

i; 

• remote message count: number of SLA bid 
scheduling message overhead for remote jobs at a 
resource i. 
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4.4 Results and observations 



4.4.1 Federation perspective 

In experiment 1, we measure how the varying of the 
total time for SLA bids coupled with Greedy backfill- 
ing resource allocation strategy affects the Grid partic- 
ipants across the federation. We quantify how the ad- 
ditional decision making time given to the LRMSes be- 
fore finalizing the SLA contracts affects the overall sys- 
tem performance in terms of resource owner's and end- 
user's objective functions. We observed that when the 
LRMSes across the federation applied FCFS technique 
for finalizing the SLAs (i.e. no decision making time 
was given, so the LRMSes have to reply as soon as 
the SLA request was made), the resource owner's made 
4.102 x 10 9 grid dollars as incentive (refer to Fig|4ja)). 

We observed that with an increase in the total SLA 
bidding time (i.e. as the LRMSes were allowed deci- 
sion making time before finalizing the SLAs hence they 
applied Greedy backfilling scheduling on the queue of 
SLA bids), the resource owners earned more incentive 
as compared to FCFS case. When 10% of total dead- 
line was allowed for SLA bids, the total incentive earned 
across the federation increased to 4.219 x 10 9 grid dol- 
lars. While, in case 50% of total job deadline was al- 
lowed for the SLA bids, the total incentive accounted 
to 4.558 x 10 9 grid dollars. Hence, the resource own- 
ers across federation exprienced an increase of approxi- 
mately 10% in their incentives as compared to the FCFS 
case. 

However, we observed that with an increase in the 
total SLA bid delay, the end-users across the federa- 
tion experienced degraded QoS. During the FCFS case, 
the average response time across the federation was 
1.183 x 10 4 sim units (refer to FigOb)). However, in 
case of 10% SLA bid delay the average response time in- 
creased to 1.344 x 10 4 sim units. Finally, when 50% of 
the total job deadline was allowed as SLA bid delay the 
average response time further increased to 1.956 x 10 4 
sim units. Furthermore, in this case the end-users end up 
spending more budget as compared to the FCFS case (re- 
fer to FigEtc)). 

Hence, we can see that although the proposed ap- 
proach leads to better optimization of resource owners' 
payoff function, it has degrading effect on the end-user's 
QoS satisfaction function across the federation. 
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4.4.2 Resource owner perspective 

In experiment 1, we quantified how varying of total SLA 
bid time/delay affects the individual resource owners 
in the federation. We analyzed, how the proposed ap- 
proach affects the superscheduling parameters related to 



(c) total SLA bid delay vs. total earnings per processor 

Figure 5. Resource owners perspective 
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the resource owner's payoff function. The most time- 
efficient resources in the federation i.e. NASA-iPSC, 
SDSC-SP2, Kth-SP2 and CTC-SP2 (refer to Tabled 
experienced substantial increase in the total incentive 
earned with an increase in total decision making time. 
When no time was allowed for decision making (FCFS 
case), these resources earned 1.764 x 10 6 , 1.61 x 10 6 , 
1.458 x 10 6 , 1.377 x 10 6 and 9.464 x 10 5 grid dol- 
lars (refer to Fig. ISJc)) P er processor. When the jobs 
in the system were allowed 30% of their total deadline 
as SLA bid time or admission control decision making 
time, these resources earned 1.946 x 10 6 , 1.957 x 10 6 , 
1.799 x 10 6 , 1.622 x 10 6 and 9.996 x 10 5 grid dollars 
per processing unit. Same trends can be observed in the 
plots for total earnings (refer to Fig. 0b)) and number 
of machine instructions executed during the simulation 
period (refer to Fig. [2 a)). 

Thus, we can see that when LRMSes are given deci- 
sion making time, they have better control over resource 
allocation/admission control decision. Furthermore, we 
can see that Greedy backfilling approach leads to better 
optimization of owner's payoff function as compared to 
the FCFS approach. 

4.4.3 End-users perspective 

In experiment 1, we also quantified the QoS satisfac- 
tion parameters for end-user's across all the resources 
in the Grid-Federation. When LRMSes across the fed- 
eration applied FCFS scheduling, end-users at the re- 
source NASA-iPSC experienced 1.719 x 10 3 sim units 
as average response time (refer to Fig|6ja)). They also 
spent 1.143 x 10 D grid dollars on the average to get their 
job done in the federation (refer to FigHJb)). However, 
when the user's allowed 50% of the total job deadline as 
SLA bid time, the average response time at NASA-iPSC 
increased to 3.170 x 10 3 sim units. In this case, end- 
users paid 1.14928 x 10 5 grid dollars. Fig[6jc) depicts 
the plot for number of jobs accepted for users across re- 
sources in the federation with increasing SLA bid time. 

Thus, we can see that FCFS based LRMS SLA con- 
tract allocation approach is better as far as end-user's 
QoS satisfaction is concerned as compared to Greedy 
backfilling. However, such an approach is difficult to re- 
alize into today's Internet based system where resource 
owners have rational goals and focus on maximizing 
their payoff function, while delivering an acceptable 
level of QoS to the end-users. 
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(b) total SLA bid delay vs. average budget spent (grid dollars) 
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4.4.4 System message complexity perspective 

In experiment 2, we quantified the message complexity 
involved with our proposed superscheduling approach. 
We measure the number of SLA bid messages required 



(c) total SLA bid delay vs. no. of jobs accepted 



Figure 6. End-users perspective 
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SLA bid delay (% of Allowed Deadline) 



(a) total SLA bid delay vs. average no. of messages per job 
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(b) total SLA bid delay vs. no. of remote messages 



on average across federation to schedule a job. This met- 
ric also includes the messages for sending the executable 
and receiving the output. Fig0a), (b) and (c) depicts the 
plots for scheduling message complexity involved with 
our approach. 

Our simulations show that when no SLA bid de- 
lay was allowed, the average SLA bid message per job 
across federation was 9.12 (refer to Fig0a)). As the 
system allowed 40% of total job deadline as SLA bid 
delay, the SLA message/job remained almost the same 
at about 9.32. Thus, we can see that our proposed super- 
scheduling approach does not incur any additional com- 
munication overhead. 

In Fig0b), we quantify the remote superscheduling 
message complexity at various resources in the Grid- 
Federation. We observed that the most time-efficient re- 
source i.e. NASA-iPSC received the maximum number 
of remote messages followed by SDSC-SP2 and KTH- 
SP2. The same characteristic holds for all cases i.e. as 
the total SLA bid time increases from 0% to 50% of the 
allowed job deadline. 

In Fig[7lc), we quantify the local superscheduling 
message complexity at various resources in the Grid- 
Federation. Results show that the resources LANL- 
Origin and LANL-CM5 were subjected to maximum 
local superscheduling messages. Both resources are 
cost-efficient and all their local users are seeking OFT. 
Hence these resources undertook SLA bid negotiation 
with time-efficient resources hence causing large num- 
ber of superscheduling messages (note that number of 
jobs at resource LANL-Origin and LANL-CM5 were 
1706 and 1287). 
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(c) total SLA bid delay vs. no. of local messages 

Figure 7. System message complexity per- 
spective 



5 Related work 

In this section, we briefly summarize the Grid su- 
perscheduling approaches that applies SLA-based or 
negotiation-based job scheduling process. 

The work in |32| proposes a multi-agent infras- 
tructure that applies SLA protocol for solving the 
Grid superscheduling problem. The SLA negotia- 
tion protocol is based on the Contract Net Proto- 
col 1 4-0 1 . The system models three types of agents: User 
agent (UA), Local Scheduler agent (LSA), and Super- 
scheduler agent (SSA). Every active site in the system 
instantiates these agents. The UAs are the resource con- 
sumers that submits jobs to SSA for execution on the 
grid platform. The UA also specifies SLA based QoS 
parameter such as expected response time, budget and 
preferred host associated with the job. The LS func- 
tionality is similar to LRMS, managing job execution 
within a administrative domain. LSA obtains job from 
the SSA that are submitted by local and remote UAs. 
The SSA agents are responsible for coordinating job su- 
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perscheduling across different site in the system. The 
SSA agents negotiates SLA parameters with the local 
LSA and remote SS before scheduling the job. The 
model defines two kinds of SLAs: Meta-SLA and Sub- 
SLA. Meta-SLA refers to the initial SLA parameters 
submitted by the UA to its SSA. A Meta-SLA presents 
high-level job requirements and it can be refined during 
negotiation process with the SSA. While the Sub-SLA 
refers to the SLA parameters that are negotiated between 
SSA and remote site SSA. The SSA decomposes the 
Meta-SLAs to form Sub-SLAs. The Sub-SLA can con- 
tain much low-level resource description such as amount 
of physical memory required, number of processors re- 
quired. In contrast we propose (i) SLA-based coordina- 
tion scheme based on computational economy; and (ii) 
our work considers site autonomy issues, and proposes 
Greedy-backfilling resource allocation for a LRMS to 
maximize resource provider payoff function. 

The work in |44| presents a grid superscheduling 
based on multiple job SLA negotiation scheme. The key 
factor motivating this work is redundantly distributing 
job execution requests to multiple sites in the grid in- 
stead of just sending to most lightly loaded one. The 
authors argue that placing job in the queue at multi- 
ple sites increases the probability that backfilling strat- 
egy will be more effective in optimizing scheduling pa- 
rameters. The superscheduling parameters include re- 
source utilization and job average turn around time. In 
other words, the scheduling parameters are system cen- 
tric. The LRMSes at various grid sites apply FCFS pol- 
icy with Easy backfilling approach for resource alloca- 
tion. Further, the system proposes dual queuing sys- 
tem at each site. One queue for local jobs while other 
queue for remote jobs. During easy backfilling the lo- 
cal job queue is given priority over remote job queue. 
In contrast to this superscheduling system, our approach 
differs in the following: (i) the job-migration or SLA- 
based coordination is based on the user centric schedul- 
ing parameters; (ii) our approach gives a LRMSes more 
flexibility over resource allocation decision; and (iii) our 
cluster resource allocation mechanism i.e. Greedy back- 
filling algorithm focuses on maximizing resource own- 
ers payoff function. 

The work in 1 38 1 models a grid superscheduler archi- 
tecture. Each grid site has a grid scheduler (GS), grid 
middleware (GM) and a local scheduler (LRMS). Three 
different coordinated superscheduling scheme is pre- 
sented for distributed load-balancing. These algorithms 
are referred to as (i) Sender-Initiated (S-I); (ii) Receiver- 
Initiated (RI); and (iii) Symmetrically-Initiated (Sy-I). 
The S-I coordination scheme is based on the informa- 
tion pull model, the GS sends a SLA negotiation enquiry 
message to all GSes in the system through its GM. Fol- 
lowing this all contacted GSes reply with he expected 



average waiting time (AWT), expected run time (ERT) 
for the requested job and current resource utilization sta- 
tus (RUS). After receiving the response, the site which 
guarantees minimum TC (turn around cost) is preferred 
for job-migration. TC is computed as the sum of AWT 
and ERT. However, in case two GS have the same value 
for TC, then RUS is utilized as a tie-breaker. While the 
R-I approach is based on information push model. Every 
GS periodically checks its own RUS at time interval a. 
if the RUS is below a certain predefined threshold S then 
the GS volunteers itself for job-migration. It broadcasts 
its RUS parameter to all GSes in the system. In case, a 
GS needs to migrate its local job then it initiates S-I base 
job migration with the volunteer nodes. Finally, the Sy-I 
approach works in both active and passive mode. Un- 
der this approach both S-I and R-I based job migration 
algorithm can be initiated by the GSes in the system. Ef- 
fectively, the information coordination is based on com- 
plete broadcast communication approach that may gen- 
erate a large number of network messages. Such ap- 
proaches has serious scalability concerns. Further, each 
GS in the system allocates resources to the remote and 
local jobs in the FCFS manner without considering any 
site-specific objective function. In contrast to this su- 
perscheduling system, our approach differs in the fol- 
lowing: (i) the SLA coordination in Grid-Federation is 
based on one-to-one SLA negotiation mechanism hence 
effectively limiting the network communication over- 
head; and (ii) we apply Greedy backfilling approach at 
grid sites for maximizing resource owner payoff func- 
tion. 

The work in [XI] presents a superscheduling sys- 
tem that consists of Internet- wide Condor work pools. 
They utilize Pastry routing substrate to organize and in- 
dex the Condor work pool. Various condor pools coor- 
dinate load-balancing information by sending resource 
status query to all the resources in the routing table. Ef- 
fectively, the query is broadcasted to all the pools that 
are indexed by the routing table. The resource status 
query (such as queue lengths and average pool utiliza- 
tion) includes enquiry about a pool's resource availabil- 
ity status and its willingness to accept remote jobs. Con- 
tacted pool manager reply with the requested info. A su- 
perscheduling manager or pool manager in the flock pe- 
riodically compares the metrics such as queue lengths, 
average pool utilization and resource availability sce- 
nario, and based on these statistics a sorted list of pools 
from most suitable to least suitable is formulated. Using 
this list, the pool manager chooses appropriate pools for 
flocking. Further, a condor work pool accepts a remote 
job if it has free resources. The issues related to site spe- 
cific resource allocation policy is not considered. In con- 
trast: (i) we consider the site autonomy issues through 
Greedy backfilling LRMS scheduling approach; (ii) our 
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SLA bidding approach gives resource owner more con- 
trol before finalizing the SLA agreements; (iii) we con- 
sider one-to-one SLA coordination mechanism for su- 
perscheduling, hence largely limiting the network com- 
munication overhead; and (iv) our approach incorpo- 
rates an economic mechanism for superscheduling. 

Tycoon |28| is a distributed market-based resource 
allocation system. Job scheduling and resource alloca- 
tion in Tycoon is based on decentralized isolated auc- 
tion mechanism. Every resource owner in the system 
runs its own auction for his local resources. Further- 
more, auctions are held independently, thus clearly lack- 
ing any coordination. Tycoon system relies on central- 
ized Service Location Services (SLS) for index resource 
auctioneers' information. Auctioneers register their sta- 
tus with the SLS every 30 seconds. In case, a auction- 
eer fails to update its information within 120 seconds 
then SLS deletes its entry. Application level supersched- 
ulers contact the SLS to gather information about vari- 
ous auctioneers in the system. Once this information is 
available, the superschedulers (on behalf of users) issue 
bids for different resources (controlled by different auc- 
tions) constraint to resource requirement and available 
budget. In this setting, various superschedulers might 
end up bidding for small subset of resources while leav- 
ing other underutilized. In other words, superscheduling 
mechanism clearly lacks coordination. A resource bid is 
defined by the tuple (h, r, b, t) where h is the host to bid 
on, r is the resource type, b is the number of credits to 
bid, and t is the time interval over which to bid. Auc- 
tioneers determine the outcome by using bid-based pro- 
portional resource sharing economy model. In contrast: 
(i) our superscheduling approach is based on decentral- 
ized commodity markets; and (ii) we consider a Greedy 
backfilling resource allocation heuristic for LRMSes. 

6 Conclusion and future work 

In this paper, we presented an SLA-based super- 
scheduling approach based on contract net protocol. The 
proposed approach models set of resource providers as 
a contract net while job superschedulers work as man- 
agers, responsible for negotiating SLA contracts and job 
superscheduling in the net. Superschedulers bid for SLA 
contracts in the net with focus on completing the job 
within the user specified deadline. We analyzed how 
the varying degree of SLA bidding time (i.e. admission 
control decision making time for LRMSes) affects the 
resource providers' payoff function. Results show that 
the proposed approach gives resource owners finer con- 
trol over resource allocation decisions. However, results 
also indicate that proposed approach has degrading ef- 
fect on the user's QoS satisfaction. However, we need to 
do more research on abstracting the user's QoS require- 



ment. We need to analyze how the deadline type for the 
user jobs can be abstracted into different types such as 
into urgent and relaxed deadline. In these cases, jobs 
with urgent requirement can be given preferences while 
finalizing SLA contracts hence providing improved QoS 
satisfaction to users. 

We analyzed how the varying bid time for SLA con- 
tracts affects the system scalability and performance in 
terms of total message complexity. In general, the pro- 
posed superscheduling heuristic does not incur exces- 
sive messages on per job basis as compared to the FCFS 
case. In our future work we will study to what ex- 
tent the user profile can change and how pricing polices 
for resources leads to varied utility of the system. We 
also intend to look into simultaneously bidding for SLA 
contracts at multiple contractor in the net, for a super- 
scheduling iteration I for a job Ji.j,k- This approach can 
increase the end-user's QoS satisfaction in terms of re- 
sponse time. As in this case the total waiting time per 
SLA bid is greatly reduced. 
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